Skip to content

refact import transformers and make cache dir alap#290

Merged
darkrush merged 4 commits intoccprocessor:devfrom
darkrush:model_interface
Mar 11, 2025
Merged

refact import transformers and make cache dir alap#290
darkrush merged 4 commits intoccprocessor:devfrom
darkrush:model_interface

Conversation

@darkrush
Copy link
Copy Markdown
Collaborator

@darkrush darkrush commented Mar 11, 2025 •
edited
Loading

Use function import_transformer to make sure import transformers after set envvar HF_HOME.

def import_transformer():
    os.environ['HF_HOME'] = CACHE_DIR
    os.makedirs(CACHE_DIR, exist_ok=True)
    import transformers

    return transformers

Now, cache dir will not be maked at importing resource_utils but at downloading or unziping.

@darkrush darkrush changed the title refact refact import transformers and make cache dir alap Mar 11, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 11, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Impacted file tree graph

@@            Coverage Diff             @@
##              dev     #290      +/-   ##
==========================================
- Coverage   89.92%   89.58%   -0.35%     
==========================================
  Files          85       82       -3     
  Lines        5630     5548      -82     
==========================================
- Hits         5063     4970      -93     
- Misses        567      578      +11
Files with missing lines Coverage Δ
llm_web_kit/model/html_classify/model.py 100.00% <100.00%> (ø)
llm_web_kit/model/policical.py 54.43% <100.00%> (-0.57%) ⬇️
llm_web_kit/model/porn_detector.py 74.11% <100.00%> (ø)
...lm_web_kit/model/resource_utils/download_assets.py 95.83% <100.00%> (+0.04%) ⬆️
llm_web_kit/model/resource_utils/unzip_ext.py 95.83% <100.00%> (+0.08%) ⬆️
llm_web_kit/model/resource_utils/utils.py 96.55% <100.00%> (+0.12%) ⬆️

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@darkrush darkrush merged commit cb1d052 into ccprocessor:dev Mar 11, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@darkrush